Dynamic Watering Point Localization for Soil Channeling Prevention Using Computer Vision¶

Teng Tian

In [ ]:
%matplotlib inline
#import library
from plantcv import plantcv as pcv
import cv2 as cv
import numpy as np
from matplotlib import pyplot as plt
#import time library to measure the execution time
import time
import math
In [ ]:
# automaticly show image after every process
pcv.params.debug="plot"

The images used in this work were captured by the on-board camera of Farmbot and were taken in the University Greenhouse.

In [ ]:
# read an image
img,path,filename=pcv.readimage(filename="../data/CAM_greenhouse/fotos_ideal_condition/Farmbot_GH_ideal_11.jpg", mode="rgb")
No description has been provided for this image

Step 1: finde the location of the plants¶

Analyzing the image in different color spaces provides a clear understanding of the image properties.

In [ ]:
pcv.params.text_size = 8
pcv.params.text_thickness = 10
colorspaces=pcv.visualize.colorspaces(rgb_img=img, original_img=False)
No description has been provided for this image

The H channel refers to the color of objects in the image. I used the equalization function to make the difference easily visible to the human eye. In the H channel, all the leaves of the plant have the same color value. The problem is that the soil particles also have some pigments with the same color value as the leaves, so they produce a lot of noise in the background. If we apply a binary threshold to this image, we can see this noise.

In [ ]:
img_H=pcv.rgb2gray_hsv(rgb_img=img, channel="H")
img_H_hist_EQU=pcv.hist_equalization(img_H)
pcv.visualize.histogram(img=img_H)
img_H_thresh, __ = pcv.threshold.custom_range(img=img_H, lower_thresh=[20], upper_thresh=[40], channel='gray')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

Using channel A will provide the most significant color difference between plants and the soil beneath. The plants appear to have a darker color.

In [ ]:
img_A=pcv.rgb2gray_lab(rgb_img=img, channel="A")
img_A_hist_EQU=pcv.hist_equalization(img_A)
pcv.visualize.histogram(img=img_A)
pcv.visualize.histogram(img=img_A_hist_EQU)
img_A_thresh, __ = pcv.threshold.custom_range(img=img_A_hist_EQU, lower_thresh=[0], upper_thresh=[30], channel='gray')
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

The V channel refers to the brightness of the objects. This can be used to eliminate all white spots caused by reflections from metal surfaces. We can also eliminate the dark background, which is the pallet holding the flower pots.

In [ ]:
pcv.params.debug="none"
img_V=pcv.rgb2gray_hsv(rgb_img=img, channel="V")
img_V_thresh_up = pcv.threshold.binary(gray_img=img_V, threshold = 250, object_type='dark')
img_V_thresh_down = pcv.threshold.binary(gray_img=img_V, threshold = 50, object_type='light')
img_V_thresh = cv.bitwise_and(img_V_thresh_up,img_V_thresh_down)
pcv.params.debug="plot"

Let's put all of our masks from the H, A, and V channels together.

In [ ]:
# in this way we habe now a binary mask
pcv.params.debug="none"

pcv.plot_image(img_H_thresh)
pcv.plot_image(img_A_thresh)
pcv.plot_image(img_V_thresh)

pcv.params.debug="plot"
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

To reduce noise in the H and A channel, we can use an erode method. Afterward, the H and A masks can be combined by performing an OR-operation. The dark objects and bright spot can be eliminated by performing an AND-operation with the V mask. As a result, the noise is mostly reduced.

In [ ]:
img_H_thresh_erode = pcv.erode(gray_img = img_H_thresh, ksize = 5, i = 1)
img_A_thresh_erode = pcv.erode(gray_img = img_A_thresh, ksize = 5, i = 1)
img_thresh = cv.bitwise_or(img_A_thresh_erode,img_H_thresh_erode)
img_thresh = cv.bitwise_and(img_thresh,img_V_thresh)
pcv.plot_image(img_thresh)
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

Using the closing method to improve the accuracy of the region-finding method.

In [ ]:
mask_dilated = pcv.dilate(gray_img = img_thresh, ksize = 5, i = 2)
mask_erode = pcv.erode(gray_img = mask_dilated, ksize = 5, i =3)
mask_dilated = pcv.dilate(gray_img = mask_erode, ksize = 5, i = 3)
mask = mask_dilated
print(np.shape(mask))
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
(960, 1280)
In [ ]:
# labeled the regions on the mask image
labeled_mask, num_mask = pcv.create_labels(mask=mask)
#pcv.plot_image(labeled_mask)
print('total', num_mask, 'region(s) found!')
No description has been provided for this image
total 47 region(s) found!

The small regions that were found are more likely a result of remaining noise in the background. In this case, it would be acceptable to only consider the larger regions.

In [ ]:
# just keep the first 10 biggst region on the mask
count = 0
region_info={}
for region_id in range(1,num_mask+1,1):
    mask_region_cnt = cv.inRange(labeled_mask,region_id,region_id)
    count = cv.countNonZero(mask_region_cnt)
    region_info[region_id]= (region_id, count)
list_of_region = list(region_info.values())
sorted_data = sorted(list_of_region, key=lambda x: x[1], reverse=True)
sorted_data_cop = sorted_data[:10]

mask_cop = np.zeros(np.shape(mask),dtype=np.uint8)
for region_id in sorted_data_cop:
    id = (int)(region_id[0])
    mask_cop+=cv.inRange(labeled_mask,id,id)

pcv.plot_image(mask_cop)
No description has been provided for this image
In [ ]:
# just keep the biggst region on the mask
'''
region_info={}
count_prev = 0
count = 0
for region_id in range(1,num_mask+1,1):
    mask_region_cnt = cv.inRange(labeled_mask,region_id,region_id)
    count = cv.countNonZero(mask_region_cnt)
    contours_draw, hierarchy = cv.findContours(mask_region_cnt,cv.RETR_EXTERNAL,cv.CHAIN_APPROX_SIMPLE)
    arcLength_contour = cv.arcLength(contours_draw[0], True)
    if arcLength_contour > 5000: # This depends on the flower pot
        next
    elif count>count_prev:
        mask_cop = mask_region_cnt
        count_prev = count
    else:
        next
pcv.plot_image(mask_cop)
'''
No description has been provided for this image
In [ ]:
# calculation the center of mass of the region
# this will locate the plant
contours, hierarchy = cv.findContours(mask_cop,cv.RETR_EXTERNAL,cv.CHAIN_APPROX_SIMPLE)
'''cnt = contours[0]
M = cv.moments(cnt)
cx = int(M['m10']/M['m00'])
cy = int(M['m01']/M['m00'])
print(cx,cy)
mask_RGB=cv.cvtColor(mask_cop,cv.COLOR_GRAY2BGR)
cv.circle(mask_RGB,(cx,cy),2,(0,255,0),20)
pcv.plot_image(mask_RGB)
'''
mask_RGB=cv.cvtColor(mask_cop,cv.COLOR_GRAY2BGR)
cv.drawContours(mask_RGB, contours, 0, (255,255,0), 3)
pcv.plot_image(mask_RGB)
No description has been provided for this image

Now we have a fairly accurate mask that shows where the plants are located.

Step 2: Determine the inner area of the flower pot¶

The hough-circle method is being used to detect flower pots in the image. As a prerequisite, it is necessary to determine the minimum distance between two pots and the pot radius in pixels. This refers to these parameters:

  • mindDist
  • minRadius
  • maxRadius
In [ ]:
# using hough-circle to dectect flower pot(s) in the image
# as a precondition, we need to know the smallest distance between two pots and the radius of the pot in pixel
'''
cv.HoughCircles (image, circles, method, dp, minDist, param1 = 100, param2 = 100, minRadius = 0, maxRadius = 0)

Parameters
image	8-bit, single-channel, grayscale input image.
circles	output vector of found circles(cv.CV_32FC3 type). Each vector is encoded as a 3-element floating-point vector (x,y,radius) .
method	detection method(see cv.HoughModes). Currently, the only implemented method is HOUGH_GRADIENT
dp	inverse ratio of the accumulator resolution to the image resolution. For example, if dp = 1 , the accumulator has the same resolution as the input image. If dp = 2 , the accumulator has half as big width and height.
minDist	minimum distance between the centers of the detected circles. If the parameter is too small, multiple neighbor circles may be falsely detected in addition to a true one. If it is too large, some circles may be missed.
param1	first method-specific parameter. In case of HOUGH_GRADIENT , it is the higher threshold of the two passed to the Canny edge detector (the lower one is twice smaller).
param2	second method-specific parameter. In case of HOUGH_GRADIENT , it is the accumulator threshold for the circle centers at the detection stage. The smaller it is, the more false circles may be detected. Circles, corresponding to the larger accumulator values, will be returned first.
minRadius	minimum circle radius.
maxRadius	maximum circle radius.
'''
img_HoughCircles = img.copy()
start_time = time.time()
circles_list = []
img_GRAY=cv.cvtColor(img,cv.COLOR_BGR2GRAY)
img_mB = cv.medianBlur(img_GRAY,5)
circles= cv.HoughCircles(image=img_mB,method=cv.HOUGH_GRADIENT,dp=4,minDist=700,param1=60,param2=40,minRadius=450,maxRadius=500)
circles = np.uint16(np.around(circles))
circles_list.append(circles)
for i in circles[0,:]:
    # draw the outer circle
    cv.circle(img_HoughCircles,(i[0],i[1]),i[2],(255,0,255),4)
    # draw the center of the circle
    cv.circle(img_HoughCircles,(i[0],i[1]),2,(0,0,255),30)
pcv.plot_image(img_HoughCircles)
end_time = time.time()
print('Execution time:', round(end_time - start_time, 2), 'seconds')
No description has been provided for this image
Execution time: 0.72 seconds

The Hough Circle algorithm may find more than one circle in the image. The circle we are searching for is among them, and this is the most crucial aspect. The next step is to select the filter to exclude incorrect circles. As a filtering criterion, we calculate the distance from the center of the circle to the center of the picture.

In [ ]:
# we take only the circle, whose center is closest to center of the image 
cnt = 1
index = 0
distance_prev = 0
cy = (int)(np.shape(img)[0]/2)
cx = (int)(np.shape(img)[1]/2)
for i in circles[0,:]:
    # draw the outer circle
    cv.circle(mask_RGB,(i[0],i[1]),i[2],(255,0,255),4)
    # draw the center of the circle
    cv.circle(mask_RGB,(i[0],i[1]),2,(0,0,255),30)
    # draw a line from the center of the circle to center of mass
    cv.line(mask_RGB,(i[0],i[1]), (cx,cy),(255,0,0),4)
    distance = (int)(math.sqrt((cx-i[0])**2+(cy-i[1])**2))
    if distance<distance_prev:
        index = cnt - 1
    print('Distance to ',cnt,'. circle =', distance)
    cnt=cnt+1
print('The',index+1, 'circle is the wanted circle')
pot_x = circles[0][index][0]
pot_y = circles[0][index][1]
pot_radius = circles[0][index][2]
print('Position of the flowerpot is', 'x=', pot_x,'y=',pot_y,'radius=',pot_radius)
pcv.plot_image(mask_RGB)
Distance to  1 . circle = 153
Distance to  2 . circle = 648
Distance to  3 . circle = 729
Distance to  4 . circle = 551
The 1 circle is the wanted circle
Position of the flowerpot is x= 714 y= 614 radius= 454
No description has been provided for this image

Step 3: Setting watering points¶

Now we know the exact location of the plants and the pot. The area in the pot can be divided for dynamic watering.

I utilized a straightforward method. An edge area was set to prevent water from flowing out of the pot and to prevent soil channels from forming at the edge of the pot. The pot is divided into 12 parts and checked for plant leaves. If there are no leaves, a watering point is set in the area.

The mask will be enlarged on purpose to create a safe zone, so that the watering point will not be set on the edge of the leaves.

In [ ]:
# let us define the watering point.
# It cannot be at the edge area, and either on top of the plant
edge_area = 150 # 100 Pixel to the edge will not be watered
# create watering point
angel = 30 # must be 15, 30, 45, 60, 90
# the previous mask will be enlarged, so that there will be a safty zone, that we will not water the leaves
mask_with_saftyzone = pcv.dilate(gray_img = mask_cop, ksize = 15, i = 3)
watering_point_list = []
for i in range(0, (int)(360/angel)):
    x_watering_point = (int)(math.cos(i*angel/180*math.pi)*(pot_radius - edge_area)+pot_x)
    y_watering_point = (int)(math.sin(i*angel/180*math.pi)*(pot_radius - edge_area)+pot_y)
    if mask_with_saftyzone[y_watering_point, x_watering_point] != 255:
        watering_point_list.append((x_watering_point, y_watering_point))
    #watering_point_list.append((x_watering_point, y_watering_point))
No description has been provided for this image
In [ ]:
# lets draw everything on image
img_out = img.copy()
cv.circle(img_out,(pot_x,pot_y),pot_radius,(255,0,255),8)
for i in watering_point_list:
    cv.circle(img_out,i,2,(0,255,0),20)
    print('prossible watering point:',i)
cv.drawContours(img_out, contours, contourIdx=-1, color=(255,0,0), thickness=3)
pcv.plot_image(img_out)
prossible watering point: (1018, 614)
prossible watering point: (977, 766)
prossible watering point: (866, 877)
prossible watering point: (714, 918)
prossible watering point: (562, 877)
prossible watering point: (450, 766)
prossible watering point: (410, 614)
prossible watering point: (450, 462)
prossible watering point: (561, 350)
prossible watering point: (714, 310)
prossible watering point: (866, 350)
prossible watering point: (977, 461)
No description has been provided for this image